With the advent of the Internet, modern computer applications not only operate on isolated systems, but also communicate with each other on data communication networks. These network applications communicate with each other by sending and receiving packets to and from each other on the networks. Network applications capture, process, and analyze the packets on the network in order to communicate with each other.
For analysis of the communication between the network applications, a specialized non-intrusive packet collection system can be deployed. For successful analysis, such a system should be able to capture, process, and analyze the packets received from other network applications in the correct order in which they were sent by the other applications. There are various commercial and open source applications performing packet analysis currently available for network applications.
The success of conventional packet analysis applications depends upon their abilities to non-intrusively capture individual data transmission packets and restore their logical Internet Protocol (IP) flows and HTTP streams. An example of a packet analysis application utilizing non-intrusive packet analysis includes, for example, the open source network intrusion detection system (SNORT). However, none of the conventional packet analysis applications are effective in HTTP reconstruction for non-intrusive capturing and analysis of packets on a high-speed network.
When inspecting a large volume of network traffic in a non-intrusive tap mode, significant challenges to address include packet drops, re-transmitted packets, and jittered packets in the reconstruction of the packet flows.
Typically, packet drops may occur in various places, including (1) at the packet provider of an auditor (in some cases, a switch with port mirroring or a network tap device), (2) at the auditor network card, (3) while processing the tapped packets. These problems are not compensated for by using the TCP reliability mechanism in a passive tap mode. The mechanisms in packet reconstruction in intrusion detection systems (IDS) also do not reconstruct entire sessions, but mostly perform inspection on a single packet or a small number of packets for signature matching.
If the TCP stream is corrupted, the HTTP application layer requests and responses can typically be reconstructed and processed into event logs. However, packet drops in the lower TCP layer may impair the content of the HTTP messages, in their headers, or in both. As a result, the HTTP stream cannot be parsed, and thus cannot be reconstructed.
It would therefore be advantageous to provide a solution that would overcome the deficiencies of the prior art solutions for capture and reconstruction of HTTP traffic.